Backtracking

ChatGPT and other LLMs have trouble with anything that involves re-processing something that has already been generated. They can’t write a palindrome, for example.

“Write a sentence that describes its own length in words”

More Examples

Raji et al. (2022): “Despite the current public fervor over the great potential of AI, many deployed algorithmic products do not work.” Although written before ChatGPT, this lengthy paper includes many examples where AI shortcomings belie the fanfare.

via Amy Castor and David Gerard: Pivot to AI: Pay no attention to the man behind the curtain

Former AAAI President Subbarao Kambhampati articulates why LLMs can’t really reason or plan

Much of the success is a “Clever Hans” phenomenon

References

Raji, Inioluwa Deborah, I. Elizabeth Kumar, Aaron Horowitz, and Andrew D. Selbst. 2022. “The Fallacy of AI Functionality.” In 2022 ACM Conference on Fairness, Accountability, and Transparency, 959–72. https://doi.org/10.1145/3531146.3533158.